Field of the disclosure
The present disclosure relates generally to video image processing and in one exemplary aspect, to methods and apparatus for the provision of rotated spherical viewports for omnidirectional videos and images.
Description of Related Art
In a so-called spherical projection, a viewing device simulates a viewing experience from within a virtualized sphere; e.g., an image is projected onto the virtual spherical surface and viewed from inside the sphere. Spherical images are typically obtained by capturing multiple images with overlapping fields of view from different cameras and combining (“stitching”) these images together. However, most viewing devices can only display a two-dimensional image. Additionally, converting a spherical projection to a two-dimensional projection format can introduce some amount of distortion and/or affect the subsequent image data. However, two-dimensional projections are desirable for compatibility with existing image processing techniques and also for most user applications. In particular, many encoders and compression techniques assume traditional rectangular image formats. Thus, projections that use so-called “fish-eye” projections are not suitable for most encoders. Similarly, some projections may drastically warp object continuity at particular regions or along certain “lines” within the image.
Virtual reality (VR) or other panoramic 360° images and video often require the rendering/display device to extract a sub-region portion of the source video/image data (a so-called “viewport”) for rendering/display purposes. One of the most common methodologies to extract a given viewport from, for example, omnidirectional video involves projecting the video data into a rectilinear projection (which is a two-dimensional projection). The conversion of omnidirectional images onto a rectilinear projection “flattens” the projection, similar to the manner in which a globe of the world is flattened when it is represented with a two-dimensional map. Rectilinear projections work well for smaller field of view (FOV) viewports (e.g., 60°×60°), but as soon as the FOV of the viewport starts to increase (e.g., 75°×75°), the corners of the viewport experience “stretching”. This pattern is exacerbated further for larger FOVs (e.g., 90°×90°). At these larger FOVs, the corners of the viewport appear extremely stretched and are aesthetically displeasing/unwatchable for a user. This is due to the geometric distortions introduced from projecting spherical data rectilinearly. For these types of projections, the point on the image that does not experience any or only minimal distortion is located at the center of the image. As points move closer to the edge (i.e., top/bottom/left/right), the stretching increases significantly.
FIG. 1 illustrates an exemplary prior art rectilinear projection (at a 75°×75° viewport) which illustrates the extreme distortions located at the corners of the image (e.g., the face in the image appears quite distorted). In FIG. 2 (prior art), the face appears with minimal distortion; however, the objects towards the edge of the image (e.g., the left-most car) appear distorted. Accordingly, prior methods for displaying VR and/or 360° panoramic content are limited as: (1) wider FOV viewports are not aesthetically pleasing (although wider FOV viewports may ostensibly give a better sense of immersion); and (2) viewport distortions may give an unnatural experience as the user dynamically moves a viewport. For example, the object may appear fine when placed in the middle of the viewport, but may be “stretched” when panning, for example, to the corner of the image. These distortions will dynamically stretch and warp the image when a user moves or “pans” the viewport.